METHOD, APPARATUS, AND PROGRAM FOR SIMILARITY JUDGMENT 

BACKGROUND OF THE INVENTION 
Field of the Invention 
5 The present invention relates to a similarity judgment 

method and a similarity judgment apparatus for judging 
similarity between images . The present invention also relates 
to a program that causes a computer to execute the similarity 
judgment method. 

10 Description of the Related Art 

In order to prevent illegal copying of image data, a 
method of finding similarity between image data of an original 
work and image data other than the original image data has been 
proposed (Japanese Unexamined Patent Publication No. 

15 11 (1999) -53541) . Furthermore, a method of searching a 
database that stores image data for finding similar image data 
has also been proposed (Japanese Unexamined Patent Publication 
No. 2000-29885). In those methods, when similarity between 
images is calculated, characteristic quantities are generated 

20 from an outline and a shape of a subject included in an image, 
position of a texture region in the image, and color and 
brightness information of the image, for example. 
Characteristic vectors are then calculated by using the 
characteristic quantities, and the similarity is judged based 

25 on inner product or distance between the characteristic 
vectors. By using such methods, illegal copies can be found 
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or a similar image can be searched for, according to similarity 
between images. 

However, in the case where similarity between images is 
judged based on characteristic quantities such as those 
5 described above, the images may not look similar in some cases 
even if the images have been judged to be similar. For example, 
if two images having people as subjects thereof have been judged 
to be similar according to the characteristic quantities, the 
images are judged to be not similar when viewed by a human 

10 observer, if the number of persons therein is not the same in 
the two images. Furthermore, in the case where images having 
dominant blue sky and dominant blue sea are compared to each 
other, the images are judged to be similar if color information 
thereof is used as the characteristic quantities. However, 

15 a human observer judges the two images to be completely 
different . 

SUMMARY OF THE INVENTION 
The present invention has been conceived based on 
consideration of the above circumstances. An object of the 
20 present invention is therefore to accurately judge similarity 
between images. 

A similarity judgment method of the present invention 
is a method of judging a similarity value between images related 
to or attached with characteristic information representing 
25 a characteristic of each of objects therein, and the similarity 
judgment method comprises the steps of: 
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calculating a similarity value between the objects 
included in the images, based on the characteristic 
information; 

calculating the similarity value between the images, 
5 based on the similarity value between the objects. 

The objects refer to subjects included in the images, 
such as a person, the sky, the sea, a tree, and a building. 

The characteristic information refers to a candidate of 
the name of each of the objects, a reliability value 
10 representing likelihood of each of the objects having the name, 
position information representing a position of each of the 
objects in a corresponding one of the images, size information 
representing a size of each of the objects, and statistic values 
such as averages or variances of characteristic quantities 
15 regarding color and brightness of each of the objects, for 
example . 

Being attached with the characteristic information 
refers to a state wherein the characteristic information is 
described in tag information of image data representing the 
20 images, for example. Being related to the characteristic 
information refers to a state wherein the characteristic 
information is organized as a file different from a file of 
the image data but not separated from the image data, for 
example . 

25 It is preferable that more than one candidate of the name 

of each of the objects to be used, since it is not known whether 
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a blue object refers to the sea or the sky, for example. 

In the similarity judgment method of the present 
invention, the images may be classified and stored according 
to the similarity value between the images. 
5 In this case, the stored images may be output 

sequentially according to the similarity value between the 
images . 

Furthermore, in this case, the images may further be 
classified and stored according to the similarity value 
10 between the objects included in the images so that the stored 
images can be output sequentially according to the similarity 
value between the objects in the images. 

A similarity judgment apparatus of the present invention 
is an apparatus for judging a similarity value between images 
15 related to or attached with characteristic information 
representing a characteristic of each of objects therein, and 
the similarity judgment apparatus comprises: 

object evaluation means for calculating a similarity 
value between the objects included in the images, based on the 
20 characteristic information; and 

image evaluation means for calculating the similarity 
value between the images, based on the similarity value between 
the objects. 

The similarity judgment apparatus of the present 
25 invention may further comprise storage means for storing the 
images in classification according to the similarity value 
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between the images. 

In this case, the similarity judgment apparatus may 
further comprise output means for sequentially outputting the 
stored images according to the similarity value between the 
5 images. 

Furthermore, the storage means may store the images by 
further classifying the images according to the similarity 
value between the objects so that the output means can 
sequentially output the stored images, based on the similarity 
10 value between the objects. 

The similarity judgment method of the present invention 
may be provided as a program to cause a computer to execute 
the method. 

According to the present invention, the similarity value 
15 between the objects in the images is calculated first based 
on the characteristic information on the objects, and the 
similarity value between the images is calculated according 
to the similarity value between the objects. Therefore, the 
similarity value between the images can be calculated based 
20 on the objects that affect impression of the images. In this 
manner, similarity between the images can be judged in a manner 
that is closer to human perception. 

By classifying and storing the images based on the 
similarity value between the images, when a predetermined one 
25 of the images is searched for or output, the images similar 
to the predetermined image can also be searched for and output 
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with ease. 

Furthermore, similarity between the images can be 
recognized with ease by outputting the images sequentially 
according to the similarity value. 
5 Moreover, if the images are further classified, stored 

and sequentially output according to the similarity value 
between the objects, any of the images including a specific 
one of the objects can be output. Therefore, the images 
including the specific object can be collected easily. 
10 BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 is a block diagram showing the configuration 
of an image classification apparatus comprising a similarity 
judgment apparatus as an embodiment of the present invention; 

Figure 2 shows characteristic information included in 
15 a metadata set; 

Figure 3 shows examples of object names; 

Figure 4 shows an example of a self-organizing map; 

Figure 5 shows specific examples of characteristic 
information; 

20 Figure 6 shows an example of a result of a search for 

similar object candidates; 

Figure 7 is a table showing a calculation result of 
evaluation values ; 

Figure 8 shows the table in which the evaluation values 
25 are sorted in descending order; 

Figure 9 shows an example of an image database; 
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Figure 10 is a flow chart showing procedures carried out 
in this embodiment; and 

Figure 11 shows another example of the image database. 
DESCRIPTION OF THE PREFERRED EMBODIMENT 
5 Hereinafter, an embodiment of the present invention will 

be explained with reference to the accompanying drawings. 
Figure 1 is a block diagram showing the configuration of an 
image classification apparatus comprising a similarity 
judgment apparatus as the embodiment of the present invention. 

10 As shown in Figure 1, the image classification apparatus 
classifies and stores image data sets Sk (k=l-n) according to 
similarity between the image data sets Sk. The image 
classification apparatus comprises image input means 1, 
metadata extraction means 2, object evaluation means 3, image 

15 evaluation means 4, image classification means 5, input means 
6, and storage means 7 . The image input means 1 comprises first 
image input means 1A and second image input means IB for 
receiving input of two of the image data sets (hereinafter 
referred to as image data sets ST1 and ST2) that are selected 

20 from the image data sets Sk as similarity evaluation targets. 
The metadata extraction means 2 comprises first metadata 
extraction means 2A and second metadata extraction means 2B 
for respectively extracting metadata sets Ml and M2 attached 
to the image data sets ST1 and ST2 . The object evaluation means 

25 3 calculates evaluation values representing similarity 
between objects included in images SGI and SG2 represented by 
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the image data sets ST1 and ST2, based on characteristic 
information included in the metadata sets Ml and M2 , as will 
be explained later. The image evaluation means 4 calculates 
a similarity value RO between the images SGI and SG2 , based 
5 on the evaluation values regarding the similarity between the 
objects found by the object evaluation means 3. The image 
classification means 5 classifies the image data sets ST1 and 
ST2 , based on the similarity value RO between the images SGI 
and SG2 . The input means 6 is used for inputting various kinds 

10 of information to the object evaluation means 3. The storage 
means 7 stores the image data sets Sk that have been classified. 
The storage means 7 is connected to a terminal 10 via a network. 

In this embodiment, the metadata sets are attached to 
the image data sets Sk. The metadata sets include the 

15 characteristic information on the objects in the images SGk 
represented by the image data sets Sk. In the case where the 
number of the objects in each of the images SGk is not one, 
the characteristic information is included in the metadata 
sets for each of the objects therein. 

20 In this embodiment, the characteristic information 

included in the metadata sets comprises items such as the name 
of each of the objects, (a first candidate name and a second 
candidate name) , a reliability value on the object name, 
position and size information of the corresponding object, and 

25 characteristic quantities extracted from pixels of the 
corresponding object, as shown in Figure 2. 



How the names of the objects are given to the objects 
in the images SGk will be explained first. Each of the images 
SGk is divided into object regions for each of the objects 
therein. The division into the object regions can be carried 
5 out according to various methods. For example, in one method, 
outlines are extracted from one of the images SGk, and a region 
surrounded by the outlines is defined as the object region. 
In another method, a region wherein brightness is constant is 
divided as the object region, based on brightness information 

10 of one of the images SGk. In still another method, a region 
of the same color is defined as the object region, according 
to color information of one of the images SGk. 

Each of the object regions is further divided into small 
regions (such as a region of 64 X 64 pixels) , and the 

15 characteristic quantities are calculated for all the small 
regions. The characteristic quantities refer to an average 
color therein and high frequency components in x and y 
directions, for example. Based on characteristic quantity 
vectors obtained from the characteristic quantities of the 

20 small regions, the name of the object to which each of the small 
regions belongs is judged. The judgment is made based on an 
output from a self -organizing map. The output is obtained by 
inputting the characteristic quantity vectors to the 
self-organizing map that has been trained regarding the names 

25 of the objects. 

The self -organizing map is obtained by training a neural 
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network comprising an input layer and an output layer having 
a plurality of units laid out two-dimensionally , and the 
characteristic quantity vectors corresponding to the object 
names are input thereto. 
5 Figure 3 shows examples of the names of the objects. As 

shown in Figure 3, the respective object names have numbers 
representing categories thereof. The number 0 represents the 
object names related to water. The numbers whose digit of ten 
is 1, such as 11 and 13, represent the object names related 

10 to the sky. The numbers in the twenties represent the object 
names related to mountains, while the numbers in the thirties 
represent the object names related to the ground. The numbers 
in the forties, the fifties, the sixties, and the seventies 
respectively represent the object names related to plants, 

15 buildings, clothes, and faces. The number 99 represents the 
object names other than those described above. 

When the self -organizing map is trained, reference 
characteristic quantity vectors used as references regarding 
the object names are input sequentially to all the units in 

20 the input layer of the neural network. The respective units 
in the input layer are connected to the respective units of 
the output layer by connection weights in accordance with the 
reference characteristic quantity vectors. The connection 
weights of one of the units having the closest fit to each of 

25 the input reference vectors (the unit is called the winner) 
are updated so as to become close to the corresponding input 
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reference vector . In this manner, the training is carried out, 
and the connection weights of the units close to the winner 
are also updated to become close to the input reference vector. 
Consequently, the neighboring units have the connection 
5 weights that are similar to each other, and a range of the units 
having the similar connection weights becomes narrower as the 
training progresses. 

Therefore, the units classified roughly in an initial 
stage of the training are classified according to the reference 

10 characteristic quantity vectors corresponding to the object 
names after repetition of the training. For example, assume 
the case where the object names are "sea", "sky", "ground", 
and "face", for the sake of simple explanation. The units 
comprising the self -organizing map are classified according 

15 to the four object names, as shown in Figure 4. In each of 
the units in the output layer, the name of the corresponding 
object and a reliability value thereof (shown as percentage) 
are set according to a result of the training. For example, 
in units Ul and U2 shown in Figure 4, the names and the 

20 reliability values are set as "sea:0:90%" and "sea:0:50%". 

When one of the characteristic quantity vectors obtained 
from the characteristic quantities of one of the small regions 
is input to the self -organizing map that has been subjected 
to the training, one of the units in the output layer having 

25 the connection weights that are closest to the input 
characteristic quantity vector responds. Since each of the 
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units in the output layer has the object name and the 
reliability value thereof according to the training result, 
the object name and the reliability value thereof are found, 
such as "sky: 10%", regarding the small region from which the 
5 input characteristic quantity vector has been obtained, 
according to a position of the unit in the output layer 
connected to the units in the input layer by the connection 
weights that are most similar to the input characteristic 
quantity vector. 

10 The object names are found for all the small regions, 

and the object name observed most frequently in the small 
regions in each of the object regions and the reliability value 
thereof become the first candidate name and the reliability 
value of the corresponding object region, that is, of the 

15 corresponding object. The object name having the 

second-largest frequency and the reliability value thereof 
become the second candidate name and the reliability value 
thereof . 

Each of the metadata sets may include not only the first 
20 and second candidate names and the reliability values thereof 
but also more candidate names and the reliability values 
thereof . 

The position information of each of the objects refers 
to barycentric coordinates of the corresponding object region 
25 or barycentric coordinates of a rectangular region to which 
the corresponding object is circumscribed. 
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The size information of the object refers to a proportion 
of the number of pixels in each of the object regions to the 
number of all pixels in the entire image, or a proportion of 
an area of the object region to an area of the entire image. 
5 The characteristic quantities of the pixels in each of 

the objects refer to averages or variances of brightness Y and 
color differences Cr and Cb of the pixels, and a proportion 
T of a texture component therein to the entire image. 
Alternatively, each of the object regions may be divided into 

10 the small regions, and the averages or the variances of 
brightness Y and color differences Cr and Cb of the small 
regions, and the proportion T of a texture component therein 
may be used as the characteristic quantities. 

The characteristic information other than the object 

15 names is included in the metadata sets after normalization 
thereof. In this example, as for the position information, 
the barycentric coordinates (x,y) of each of the objects is 
normalized by lengths of the corresponding image in the x and 
y directions. More specifically, values (x / the length in 

20 the x direction X 100, y / the length in the y direction X 
100) are calculated. As for the size information, the 
proportion of the object region to the entire image is used. 
More specifically, a value (the number of pixels in the object 
region / the number of all the pixels in the corresponding image 

25 X 100) is used as the size information. 

For the brightness Y, an average of the object brightness 
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is normalized by a maximal brightness thereof. More 
specifically, in the case where the brightness is represented 
by 8-bit data having 255 as the maximum value, the brightness 
information is calculated as a value (the average of the 
5 brightness value Y ranging from 0 to 255 / 255 X 100) . For 
the color differences Cr and Cb, averages of the color 
differences Cr and Cb are normalized by a maximal color 
difference. More specifically, in the case where the color 
differences are represented by 8-bit data, the color 
10 difference information is calculated as values (the averages 
of the color difference values ranging from -255 to 255 / 255 
X 100) . 

Figure 5 shows an example of the characteristic 
information. As shown in Figure 5, for a first object in the 

15 image SGI, the first candidate name is "sky" and the reliability 
value thereof is 70%. The second candidate name is "sea" and 
the reliability value thereof is 30%. The position 
information is "x:20%, y:25%", and the size information is 
"15%" . The characteristic quantities of the pixels are "Y : 7 0%, 

20 Cr:-20%, Cb:+25%, T:l%". 

For a third object in the image SG2, the first candidate 
name is "sky" and the reliability value thereof is "90%". The 
second candidate name is "sea" and the reliability value 
thereof is "10%" . The position information is "x: 50%, y :25%", 

25 and the size information is "30%". The characteristic 
quantities of pixels are "Y:60%, Cr:-30%, Cb:+45%, T:l%". 
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For a fourth object in the image SG2, the first candidate 
name is "sea" and the reliability value thereof is "80%". The 
second candidate name is "sky" and the reliability value 
thereof is "20%" . The position information is "x: 50%, y : 50%", 
5 and the size information is "25%". The characteristic 
quantities of pixels are "Y:45%, Cr:-20%, Cb:+25%, T:15%". 

In Figure 5, the first and second candidate names are 
shown as the names themselves, such as "sea" or "sky". However, 
the candidate names are actually represented by the numbers 
10 shown in Figure 3. 

The first and second image input means 1A and IB in the 
image input means 1 are used for inputting the image data sets 
ST1 and ST2 . The first and second image input means 1A and 
IB comprise a media drive for reading the image data sets ST1 
15 and ST2 from a recording medium, or a communication interface 
for receiving the image data sets ST1 and ST2 transferred via 
a network. 

The first and second metadata extraction means 2A and 
2B in the metadata extraction means 2 are used for extracting 
20 the metadata sets Ml and M2 attached to the image data sets 
ST1 and ST2 . 

The object evaluation means 3 evaluates the similarity 
for all combinations of the objects included in the images SGI 
and SG2, based on the characteristic information thereof 
25 included in the metadata sets Ml and M2 . The evaluation of 
the similarity is carried out by calculating the evaluation 
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values representing the similarity in the form of percentages. 
Hereinafter, the evaluation will be explained. 

Firstly, the objects in the image SG2 whose first or 
second candidate name is the same as the first candidate name 
5 of the first object in the image SGI (hereinafter referred to 
as similar object candidates) are found. For example, if the 
first candidate name of the first object in the image SGI is 
"sky", all the objects whose first or second candidate name 
is "sky" in the image SG2 are found as the similar object 

10 candidates. Since the candidate names are represented by the 
numbers as shown in Figure 3, all the similar object candidates 
can be found by comparing the number of the first candidate 
name of the first object in the image SGI with the numbers of 
the first and second candidate names of all the objects in the 

15 image SG2 . 

For example, if the characteristic information is as 
shown in Figure 5, the first candidate name of the first object 
in the image SGI is "sky" , and the first candidate name of the 
third object and the second candidate name of the fourth object 

20 in the image SG2 are also "sky". Therefore, the third and 
fourth objects in the image SG2 are the similar object 
candidates regarding the first object in the image SGI. 

The evaluation values are set to 0% for non-similar 
object candidates whose first or second candidate name is not 

25 the same as the first candidate name of the first object in 
the image SGI . 
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Likewise, the first candidate name of a kth object 
(k-l-n) in the image SGI is compared with the first and second 
candidate names of all the objects in the image SG2 , and the 
similar object candidates are found for all the objects in the 
5 image SGI. Figure 6 shows a result of the similar object 
candidates. In Figure 6, circles O and crosses X represent 
the similar object candidates and the non-similar object 
candidates for the kth object in the image SGI, respectively. 

The evaluation values are then found for all the similar 
10 object candidates for the kth object in the image SGI. The 
evaluation values are calculated as follows: 

evaluation value = (2 - distance between the 
characteristic quantities X weight coefficient) / 2 X 
reliability value CI X reliability value C2 
15 (1) 

where the reliability value CI and the reliability value 
C2 refer to the reliability value of the first candidate name 
of the kth object in the image SGI and the reliability value 
of the first or second candidate name of the similar object 
20 candidate in the image SG2 corresponding to the first candidate 
name of the kth object in the image SGI, respectively. 

The weight coefficient is used to determine which is more 
important between the distance and the reliability value, and 
is usually set to 1. The distance between the characteristic 
25 quantities is calculated by the following equation: 

distance = ( (Y1-Y2) 2 + (Crl-Cr2) 2 + (Cbl-Cb2) 2 + (T1-T2) 2 +Wa 
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X (position dif f erence) 2 +Wb (area dif f erence) 2 ) 1/2 

(2) 

In the Equation (2) , if the distance is more than 1, the 
distance is set to 1. The indices 1 and 2 of the brightness 
5 Y, the color differences Cr and Cb, and the texture T correspond 
to the images SGI and SG2, respectively. Wa and Wb are weight 
coefficients. Regardless of the positions or the sizes of the 
objects, the weight coefficients Wa and Wb can be set to a small 
number by an input from the input means 6 so that the position 

10 difference and the area difference can contribute less to 
similarity judgment in the case where the objects such as sea 
in the two images are desired to be judged similar regardless 
of the positions of the sea or the areas occupied by the sea 
in the two images, for example. 

15 The area difference is a difference between values of 

the size information in the characteristic information. The 
position difference can be calculated according to the 
equation below: 

position dif ference = ( (xl-x2 ) 2 + (yl-y2 ) 2 ) 1/2 (3) 

20 In the case where the similar object candidates have the 

same candidate name as the second candidate name of the kth 
object in the image SGI, the evaluation values are calculated 
according to Equation (1) for the second candidate name of the 
kth object. In this case, the larger value between the 

25 evaluation values for the first and second candidate names is 
used as the final evaluation value for the kth object in the 
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image SGI . 

In the case where the weight coefficients Wa and Wb are 
0 and the sky and the sea are found only in the first object 
of the image SGI and in the third and fourth objects in the 
5 image SG2, the evaluation values between the first object in 
the image SGI and the third object in the image SG2 are 55% 
and 26% regarding the sky and the sea, respectively. Therefore, 
the larger value 55% is used as the final evaluation value 
between the two objects. Likewise, the evaluation values 

10 regarding the first object in the image SGI and the fourth 
object in the image SG2 are 12% and 21% for the sky and the 
sea, respectively. Therefore, the evaluation value 21% is 
used as the final evaluation value between the two objects. 
Figure 7 shows a table TB representing a result of the 

15 calculation of the evaluation values. The object evaluation 
means 3 outputs the table TB shown in Figure 7 to the image 
evaluation means 4. 

The image evaluation means 4 judges the similarity 
between the images SGI and SG2 , based on the table TB 

20 representing the evaluation result output from the object 
evaluation means 3. Hereinafter, how the similarity is judged 
will be explained. The image evaluation means 4 sorts the 
contents of the table TB shown in Figure 7 into the table shown 
in Figure 8, in descending order of the evaluation values 

25 between the objects. The evaluation values between the 
objects are used as they are, as the evaluation values in Figure 
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8. 

The similarity value between the images SGI and SG2 is 
found according to Equation (4) below, based on a total of the 
evaluation values between the objects exceeding a reference 
5 value (40%, in this case) : 

similarity value R0 = 1- (reference value) / (the total) 

(4) 

The higher the evaluation values are (that is, the higher 
a possibility of existence of the similar objects) , the closer 

10 to 1 the similarity value R0 becomes. Furthermore, the larger 
the number of the objects having the higher evaluation values 
is, the closer to 1 the similarity value R0 becomes, since the 
term (reference value) / (the total) becomes smaller. This is 
because the similarity becomes higher in the case where the 

15 number of the objects such as sea, sky, and tree that agree 
between the images SGI and SG2 becomes larger, than in the case 
where only the object "sky" agrees between the two images SGI 
and SG2. 

In this embodiment, the similarity value R0 between the 
20 images SGI and SG2 is 0.72. In the case where the evaluation 
values between the objects do not exceed the reference value, 
the similarity value R0 becomes 0. The image evaluation means 
4 outputs the similarity value R0 between the two images to 
the image classification means 5. 
25 The image classification means 5 compares the similarity 

value R0 input from the image evaluation means 4 with a 
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predetermined image similarity range. In the case where the 
similarity value RO is within the image similarity range, the 
images SGI and SG2 are classified as similar images. The image 
classification means 5 then classifies the image data sets ST1 
5 and ST2 according to the classification, and stores the image 
data sets ST1 and ST2 in the storage means 7 together with the 
classification result . 

The procedures described above are repeated for all 
combinations of the image data sets Sk, and the similar images 

10 are classified in the same category and stored in the storage 
means 7. The storage means 7 has an image database DB1 
regarding the image data sets Sk. Figure 9 shows an example 
of the image database. As shown in Figure 9, the image database 
DB1 has information on the file name, the time of generation, 

15 a size, properties, and the similar images, regarding each of 
the image data sets Sk stored in the storage means 7. The 
similar images are stored in order of higher similarity value 
RO between the image represented by each of the image data set 
Sk and the images similar thereto. For example, as shown in 

20 Figure 8, the similar images for the image data set ST1 are 
the images represented by the image data sets ST4, ST7 and ST3, 
in the order of higher similarity. 

The operation of the embodiment will be explained next. 
Figure 10 is a flow chart showing procedures carried out in 

25 this embodiment. The procedures of classifying the image data 
sets according to the similarity value will be explained below. 
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The eombi ricttion of the image data sets StI and ST2 to iDe 
subjected to similarity judgment is determined, among the 
image data sets Sk (Step SI) . The two image data sets are then 
input to the first and second image input means 1A and IB (Step 
5 S2) . The first and second metadata extraction means 2A and 
2B extract the metadata sets Ml and M2 from the image data sets 
STI and ST2 (Step S3) , and inputs the metadata sets Ml and M2 
to the object evaluation means 3. 

The object evaluation means 3 calculates the evaluation 

10 values between the objects, based on the characteristic 
information of the objects in the metadata sets Ml and M2 (Step 
S4), and generates the table TB shown in Figure 7 (Step S5) . 
The table TB is input to the image evaluation means 4, and the 
image evaluation means 4 calculates the similarity value R0 

15 between the images (Step S6) . The similarity value R0 is input 
to the image classification means 5, and the image 
classification means 5 classifies the image data sets STI and 
ST2 , based on the similarity value R0 (Step S7) . 

Whether the classification has been carried out for all 

20 the combinations of the image data sets Sk is then judged (Step 
S8) . If a result at Step S8 is negative, the combination is 
newly determined (Step S9) , and the process returns to Step 
S2 . If the result at Step S8 is affirmative, the image database 
DB1 is updated according to the classification result (Step 

25 S10) to end the process. 

In the case where an output instruction is input from 
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the terminal 10 connected to the storage means 7 via the network 
regarding one of the image data sets Sk stored in the storage 
means 7, the image database DB1 is referred to, and the similar 
image data sets are output to the terminal 10 in order of higher 
5 similarity, together with the image data set of the output 
instruction. For example, if the image database DB1 is as 
shown in Figure 8 and the terminal 10 inputs the output 
instruction regarding the image data set ST1, the image data 
sets S4, S7, and S3 are output in this order together with the 

10 image data set ST1 to the terminal 10. 

In the case where the terminal 10 searches for a portion 
of the image data sets Sk stored in the storage means 7 according 
to a predetermined search condition, the image database DB1 
is referred to, and the image data sets similar to the searched 

15 one of the image data sets are output in order of higher 
similarity, together with the searched image data set. For 
example, in the case where the image database DB 1 is as shown 
in Figure 8, if the searched one of the image data sets is ST1, 
the image data sets S4, S7, and S3 are output in this order 

20 to the terminal 10, together with the image data set ST1 . 

As has been described above, in this embodiment, the 
evaluation values are calculated first between the objects in 
the images SGI and SG2, based on the characteristic information 
on the objects, and the similarity value R0 is found for the 

25 images SGI and SG2, based on the evaluation values between the 
objects. Therefore, the similarity between the images can be 
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calculated based on the objects that affect impression of the 
images, and the similarity can be judged in a manner closer 
to human sensation. 

By storing the image data sets Sk in classification, the 
5 image data sets similar to a predetermined one of the image 
data sets can also be output or searched for with ease in the 
case where the predetermined image data set is output or 
searched for from the image data sets stored in the storage 
means . 

10 By classifying the image data sets Sk according to the 

similarity value between the images and by sequentially 
outputting the image data sets similar to the predetermined 
image data set in the case of output instruction of the 
predetermined image data set, the similarity between the 

15 images can be easily recognized. 

In the above embodiment, the image database DB1 may store 
information on the objects in each of the images and on the 
image data sets whose evaluation values with the objects are 
high, as shown in Figure 11. In this case, the terminal 10 

20 connected to the storage means 7 enables selection of the 
objects in the output image data set. In the case where the 
image database DB1 is as shown in Figure 11, if the output 
instruction is input regarding the image data set ST1 and the 
object 1 is selected from the objects include in the image 

25 represented by the image data set ST1, the image data sets ST2, 
S9, and S7 having the high evaluation values with the object 
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1 are output in this order to the terminal 10. In this manner, 
the terminal 10 can easily collect the images having a 
predetermined one of the objects such as sea. 

In the above embodiment , an image data set may be input 
5 from the terminal 10 so that the image data sets similar to 
the input image data set can be searched for from the image 
data sets Sk stored in the storage means 7 and output to the 
terminal 10. In this case, the similarity value R0 is 
calculated in the same manner described above, based on a 

10 metadata set of the input image data set and the metadata sets 
of the image data sets Sk stored in the storage means 7 . The 
image data sets similar to the input image data set are output 
from the storage means 7 to the terminal 10 in order of higher 
similarity value R0 . 

15 As has been described above, the similarity judgment 

method of the present invention can be applied to a search for 
image data sets similar to an input image data set. 
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